Three-dimensional sensing device and specular reflection object detection method

ABSTRACT

A three-dimensional (3D) sensing device is configured to sense an object. The 3D sensing device includes a flood light source, a structured light source, an image sensor, and a controller. The controller is configured to perform: commanding the flood light source and the structured light source to emit a flood light and a structured light in sequence; commanding the image sensor to sense a first reflective light and a second reflective light in sequence, so as to obtain a first image frame and a second image frame; combining the first image frame and the second image frame into a determination frame; and determining that the object is a specular reflection object in response to determining that the determination frame has at least two spots having gray levels satisfying a predetermined condition. A specular reflection object detection method is also provided.

BACKGROUND Technical Field

The disclosure generally relates to a sensing device and a detectionmethod and, in particular, to a three-dimensional (3D) sensing deviceand a specular reflection object detection method.

Description of Related Art

Recently, 3D sensing is widely applied to face recognition, simultaneouslocalization and mapping (SLAM), etc. Generally, 3D sensing can beclassified into time-of-flight (ToF) sensing, dual camera, andstructured light sensing. In the structured light sensing, a structuredlight is projected onto an object to form light pattern on the object,and a camera is configured to photograph the light pattern. Bycalculating the shifts of the points of the light pattern, distancesbetween the points of the light pattern and the camera are obtained.

However, if the object is made of glass, since the structured light willpass through the transparent object, the light pattern cannot be formedon the object. As a result, the distance of the object cannot becalculated, so that the system cannot know that there is glass in front.Consequently, when the 3D sensing is applied to an artificialintelligence (AI) robot or a sweeping robot, some collision may occur.

SUMMARY

Accordingly, the disclosure is directed to a 3D sensing device, whichcan detect a specular reflection object.

The disclosure is directed to a specular reflection object detectionmethod, which can detect a specular reflection object.

An embodiment of the disclosure provides a three-dimensional (3D)sensing device configured to sense an object. The 3D sensing deviceincludes a flood light source, a structured light source, an imagesensor, and a controller. The flood light source is configured to emit aflood light to the object, wherein the object reflects the flood lightinto a first reflective light. The structured light source is configuredto emit a structured light to the object, wherein the object reflectsthe structured light into a second reflective light. The image sensor isconfigured to sense the first reflective light and the second reflectivelight. The controller is electrically connected to the flood lightsource, the structured light, and the image sensor, and configured toperform: commanding the flood light source and the structured lightsource to emit the flood light and the structured light in sequence;commanding the image sensor to sense the first reflective light and thesecond reflective light in sequence, so as to obtain a first image framecorresponding to the first reflective light and a second image framecorresponding to the second reflective light; combining the first imageframe and the second image frame into a determination frame; anddetermining that the object is a specular reflection object in responseto determining that the determination frame has at least two spotshaving gray levels satisfying a predetermined condition.

An embodiment of the disclosure provides a specular reflection objectdetection method including: commanding a flood light source to emit aflood light and a structured light source to emit a structured light insequence; commanding an image sensor to sense a first reflective lightformed by an object reflecting the flood light and a second reflectivelight formed by the object reflecting the structured light in sequencein sequence, so as to obtain a first image frame corresponding to thefirst reflective light and a second image frame corresponding to thesecond reflective light; combining the first image frame and the secondimage frame into a determination frame; and determining that the objectis a specular reflection object in response to determining that thedetermination frame has at least two spots having gray levels satisfyinga predetermined condition.

In the 3D sensing device and the specular reflection object detectionmethod according to embodiment of the disclosure, the first image framecorresponding to the flood light and the second image framecorresponding to the structured light are combined into thedetermination frame, and that the object is a specular reflection objectis determined in response to determining that the determination framehas at least two spots having gray levels satisfying a predeterminedcondition. Therefore, the 3D sensing device and the specular reflectionobject detection method can find a specular reflection object in front.As a result, when the 3D sensing device and the specular reflectionobject detection method are applied to a device such as an AI robot, asweeping robot, a drone, etc., collision with a specular reflectionobject, e.g. a transparent object, may be prevented.

To make the aforementioned more comprehensible, several embodimentsaccompanied with drawings are described in detail as follows.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the disclosure, and are incorporated in and constitutea part of this specification. The drawings illustrate exemplaryembodiments of the disclosure and, together with the description, serveto explain the principles of the disclosure.

FIG. 1 is a schematic view of a three-dimensional (3D) sensing deviceaccording to an embodiment of the disclosure.

FIG. 2A shows diffuse reflection.

FIG. 2B shows specular reflection.

FIG. 3A shows a first image frame obtained by the image sensor in FIG. 1capturing the first reflective light when the object is a specularreflection object.

FIG. 3B shows a second image frame obtained by the image sensor in FIG.1 capturing the second reflective light when the object is a specularreflection object.

FIG. 3C shows a determination frame obtained by combining the firstimage frame in FIG. 3A and the second image frame in FIG. 3B.

FIG. 4 is a flowchart of a specular reflection object detection methodaccording to an embodiment of the disclosure.

FIG. 5 shows the determination frame in FIG. 3C being divided into aplurality of regions.

FIG. 6A shows a first image frame obtained by the image sensor in FIG. 1when the object is a light source.

FIG. 6B shows a second image frame obtained by the image sensor in FIG.1 when the object is a light source.

FIG. 6C shows a determination frame obtained by combining the firstimage frame in FIG. 6A and the second image frame in FIG. 6B.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the present embodiments of thedisclosure, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers are used in thedrawings and the description to refer to the same or like parts.

FIG. 1 is a schematic view of a three-dimensional (3D) sensing deviceaccording to an embodiment of the disclosure. Referring to FIG. 1 , the3D sensing device 100 in this embodiment is configured to sense anobject 50. The 3D sensing device 100 includes a flood light source 110,a structured light source 120, an image sensor 132, and a controller140. The flood light source 110 is configured to emit a flood light 111to the object 50, wherein the object 50 reflects the flood light 111into a first reflective light 51. The structured light source 120 isconfigured to emit a structured light 121 to the object 50, wherein theobject 50 reflects the structured light 121 into a second reflectivelight 53. The image sensor 132 is configured to sense the firstreflective light 51 and the second reflective light 53.

In this embodiment, the flood light 111 may uniformly irradiate theobject 50. The flood light source 110 may include a light-emittingelement 112 and a lens 114. The light-emitting element 112 is, forexample, a light-emitting diode (LED) or any other appropriatelight-emitting device. The light-emitting element 112 is configured toemit the flood light 111, and the lens 114 is disposed on a path of theflood light 111 and configured to transmit the flood light 111 to theobject 50. If the object 50 is a diffuse reflection object thatdiffusely reflects the flood light 111, a relatively uniformillumination distribution may be formed on the object 50.

In this embodiment, the structured light source 120 includes a laseremitter 122 and a diffractive optical element (DOE) 124. The laseremitter 122 is configured to emit a laser beam. The DOE 124 is disposedon a path of the laser beam, and configured to diffract the laser beaminto the structured light 121. The laser emitter 122 is, for example, alaser diode, a vertical-cavity surface-emitting laser (VCSEL), or anyother appropriate laser. If the object 50 is a diffuse reflectionobject, the structured light 121 may form a light pattern, e.g. a lightdot array, on the object 50.

The controller is electrically connected to the flood light source, thestructured light, and the image sensor. A camera 130 may include theimage sensor 132 and a lens 134 disposed in front of the image sensor132. If the object 50 is a diffuse reflection object, when the floodlight 111 irradiates the object 50, the lens 134 receives the firstreflective light 51 and forms an image of the object 50 on the imagesensor 132. As a result, the camera 130 can capture the image of theprofile of the object 50, and the controller may recognize the object50. If the object 50 is a diffuse reflection object, when the structuredlight 121 irradiates the object 50, a light pattern, e.g. a light dotarray, is formed on the object 50, and the lens 134 receives the secondreflective light 53 and forms an image of the light pattern, e.g. alight dot array, on the image sensor. The profile of the object 50 maycause distortion of the light pattern or shifts of light dots of thelight dot array. The controller 140 may calculate the distortion of thelight pattern or the shifts of the light dots of the light pattern, soas to obtain distances between the positions of the light pattern andthe camera. Therefore, the 3D sensing device 100 can perform 3D sensingin this way.

FIG. 2A shows diffuse reflection, and FIG. 2B shows specular reflection.Referring to FIG. 2A, when a light 62 irradiate a diffuse reflectionobject 50 a, the diffuse reflection object 50 a diffusely reflects thelight 62 into rays 64 a with different directions. However, referring toFIG. 2B, when a light 62 irradiate a specular reflection object 50 b,the specular reflection object 50 b specularly reflects the light into areflective light 64 b with a single specific direction. As a result,referring to FIG. 1 , if the object 50 is a specular reflection object,the first reflective light 51 is along a single direction, and thesecond reflective light 53 is along a single direction. Consequently,the first reflective light 51 forms a single spot P1 in the image framecaptured by the image sensor 132, as shown in FIG. 3A, and the secondreflective light 53 forms a single spot P2 in the image frame capturedby the image sensor 132, as shown in FIG. 3B.

FIG. 3A shows a first image frame obtained by the image sensor in FIG. 1capturing the first reflective light when the object is a specularreflection object. FIG. 3B shows a second image frame obtained by theimage sensor in FIG. 1 capturing the second reflective light when theobject is a specular reflection object. FIG. 3C shows a determinationframe obtained by combining the first image frame in FIG. 3A and thesecond image frame in FIG. 3B. FIG. 4 is a flowchart of a specularreflection object detection method according to an embodiment of thedisclosure. Referring to FIG. 1 and FIG. 4 , the specular reflectionobject detection method in this embodiment may be performed by the 3Dsensing device 100 shown in FIG. 1 . The controller 140 may beconfigured to perform the following steps, and the specular reflectionobject detection method in this embodiment may include the followingsteps. First, step S110 may be executed, in which the controller 140enters a specular reflection object detection mode. In this mode, theauto-exposure gain of the image frame obtained by the image sensor 132is low. For example, in an embodiment, the auto-exposure gain in thespecular reflection object detection mode is set to be 1, but thedisclosure is not limited thereto.

Next, step S120 is executed, in which the flood light source 110 and thestructured light source 120 are commanded by the controller 140 to emitthe flood light 111 and the structured light 121 in sequence, and theimage sensor 132 is commanded by the controller 140 to sense the firstreflective light 51 and the second reflective light 53 in sequence, soas to obtain a first image frame (as shown in FIG. 3A) corresponding tothe first reflective light 51 and a second image frame (as shown in FIG.3B) corresponding to the second reflective light 53.

In this embodiment, step S120 may include sub-step S122 and sub-step124. In sub-step S122, the flood light source 110 is commanded by thecontroller 140 to emit the flood light 111, and the image sensor 132 iscommanded by the controller 140 to sense the first reflective light 51,so as to obtain a first image frame (as shown in FIG. 3A) correspondingto the first reflective light 51. In sub-step S124, the structured lightsource 120 is commanded by the controller 140 to emit the structuredlight 121, and the image sensor 132 is commanded by the controller 140to sense the second reflective light 53, so as to obtain a second imageframe (as shown in FIG. 3B) corresponding to the second reflective light53. In this embodiment, sub-step S122 is executed first, and then thesub-step S124 is executed.

Then, step S130 is executed, in which the first image frame (as shown inFIG. 3A) and the second image frame (as shown in FIG. 3B) is combined bythe controller 140 into a determination frame (as shown in FIG. 3C). Forexample, in step S130, difference values between gray levels of pixelsof the first image frame and gray levels of corresponding pixels of thesecond image frame are calculated by the controller 140. Specifically,gray levels of pixels of the first image frame are subtracted from graylevels of corresponding pixels of the second image frame, or gray levelsof pixels of the second image frame are subtracted from gray levels ofcorresponding pixels of the first image frame, so as to obtain thedifference values. Then, absolute values of the difference values arecalculated by the controller 140 to respectively form gray levels ofpixels of the determination frame, e.g. the gray levels of the pixels ofFIG. 3C.

After that, that the object 50 is a specular reflection object (i.e.that there is a specular reflection object in front) is determined bythe controller 140 (step S150) in response to the controller 140determining that the determination frame has at least two spots (e.g.the spots P1′ and P2′ in FIG. 3C) having gray levels satisfying apredetermined condition (step S140). In this embodiment, thepredetermined condition is that a gray level of the at least two spotsis greater than a first threshold value, for example, 200. In thisembodiment, the gray levels of pixels of the image sensor 132 range from0 to 255.

Specifically, in step S140, the determination frame is divided into aplurality of regions R1, as shown in FIG. 5 . Then, regions R1 having anaverage of gray levels less than a second threshold value are found. Thesecond threshold value may be less than the first threshold value. Inthis embodiment, the second threshold value is, for example, 50. Afterthat, that the determination frame has at least two spots having graylevels satisfying the predetermined condition is determined in responseto the controller 140 determining the number of spots having gray levelsgreater than the first threshold value is greater than or equal to athird threshold value. In this embodiment, the third threshold value is,for example, 2. Moreover, in this embodiment, the regions R1 arearranged in a rectangular array.

In this embodiment, the at least two spots are contributed by thespecular reflection object specularly reflecting the flood light 111 andthe structured light 121. The specular reflection object may include atransparent object. For example, a material of the transparent objectincludes glass. For instance, when the object 50 is made of glass, theobject 50 may specularly reflect the flood light 111 and the structuredlight 121.

If the object 50 is a light source, the light spot P3 in the first frame(as shown in FIG. 6A) and the light spot P4 in the second frame (asshown in FIG. 6B) have the same position and size. Therefore, thedifference values in the region of the light spot P3 or P4 are zero, sothat the determination frame (as shown in FIG. 6C) does not have atleast two spots having gray levels satisfying the predeterminedcondition. Consequently, the controller will not determine that there isa specular reflection object in front. On the other hand, if the object50 is a specular reflection object, since the flood light source 110 andthe structured light source 120 are disposed at difference, so that thespot P1 in the first image frame and the spot P2 in the second imageframe respectively have different positions in the image frame.Therefore, difference values between gray levels of pixels of the firstimage frame and gray levels of corresponding pixels of the second imageframe have large absolute values in the positions of the spot P1 and thespot P2. Consequently, two spots P1′ and P2′ occur in the determinationframe as shown in FIG. 3C, so that the controller 140 may determine thatthere is a specular reflection object in front. If the object 50 is adiffuse reflection object or the is no object in front, thepredetermined condition of the determination frame will not satisfied,so that the controller 140 determines that there is no specularreflection object in front.

Moreover, in this embodiment, in step S110, since the auto-exposure gainin the specular reflection object detection mode is low, for example,being set to be 1, so that the light pattern formed by the structuredlight 121 irradiating a diffuse reflection object is not obvious and canbe ignored in comparison with the spot P1 and P2 due to specularreflection and being over-exposed and having high gray level. As aresult, the specular reflection object detection mode of step S110 issuitable for specular reflection object detection.

In the 3D sensing device 100 and the specular reflection objectdetection method in this embodiment, the first image frame correspondingto the flood light 111 and the second image frame corresponding to thestructured light 121 are combined into the determination frame, and thatthe object 50 is a specular reflection object is determined in responseto determining that the determination frame has at least two spotshaving gray levels satisfying the predetermined condition. Therefore,the 3D sensing device 100 and the specular reflection object detectionmethod can find a specular reflection object in front. As a result, whenthe 3D sensing device 100 and the specular reflection object detectionmethod are applied to a device such as an AI robot, a sweeping robot, adrone, etc., collision with a specular reflection object, e.g. atransparent object, may be prevented. Moreover, the 3D sensing device100 may detect a specular reflection object by the optical systemthereof without adopting an additional sensor of other types, e.g. anultrasonic sensor, so that the 3D sensing device 100 may have a lowercost.

In conclusion, in the 3D sensing device and the specular reflectionobject detection method according to embodiment of the disclosure, thefirst image frame corresponding to the flood light and the second imageframe corresponding to the structured light are combined into thedetermination frame, and that the object is a specular reflection objectis determined in response to determining that the determination framehas at least two spots having gray levels satisfying a predeterminedcondition. Therefore, the 3D sensing device and the specular reflectionobject detection method can find a specular reflection object in front.As a result, when the 3D sensing device and the specular reflectionobject detection method are applied to a device such as an AI robot, asweeping robot, a drone, etc., collision with a specular reflectionobject, e.g. a transparent object, may be prevented.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed embodimentswithout departing from the scope or spirit of the disclosure. In view ofthe foregoing, it is intended that the disclosure covers modificationsand variations provided that they fall within the scope of the followingclaims and their equivalents.

What is claimed is:
 1. A three-dimensional (3D) sensing deviceconfigured to sense an object, the 3D sensing device comprising: a floodlight source configured to emit a flood light to the object, wherein theobject reflects the flood light into a first reflective light; astructured light source configured to emit a structured light to theobject, wherein the object reflects the structured light into a secondreflective light; an image sensor configured to sense the firstreflective light and the second reflective light; and a controllerelectrically connected to the flood light source, the structured light,and the image sensor, and configured to perform: commanding the floodlight source and the structured light source to emit the flood light andthe structured light in sequence; commanding the image sensor to sensethe first reflective light and the second reflective light in sequence,so as to obtain a first image frame corresponding to the firstreflective light and a second image frame corresponding to the secondreflective light; combining the first image frame and the second imageframe into a determination frame; and determining that the object is aspecular reflection object in response to determining that thedetermination frame has at least two spots having gray levels satisfyinga predetermined condition.
 2. The 3D sensing device according to claim1, wherein combing the first image frame and the second image frame intothe determination frame comprises: calculating difference values betweengray levels of pixels of the first image frame and gray levels ofcorresponding pixels of the second image frame; and calculating absolutevalues of the difference values to respectively form gray levels ofpixels of the determination frame.
 3. The 3D sensing device according toclaim 1, wherein the predetermined condition is that a gray level isgreater than a first threshold value.
 4. The 3D sensing device accordingto claim 3, wherein determining that the determination frame has atleast two spots having gray levels satisfying the predeterminedcondition comprises: dividing the determination frame into a pluralityof regions; finding regions having an average of gray levels less than asecond threshold value; and determining that the determination frame hasat least two spots having gray levels satisfying the predeterminedcondition in response to determining the number of spots having graylevels greater than the first threshold value is greater than or equalto a third threshold value.
 5. The 3D sensing device according to claim4, wherein the third threshold value is
 2. 6. The 3D sensing deviceaccording to claim 4, wherein the regions are arranged in a rectangulararray.
 7. The 3D sensing device according to claim 1, wherein the atleast two spots are contributed by the specular reflection objectspecularly reflecting the flood light and the structured light.
 8. The3D sensing device according to claim 7, wherein the specular reflectionobject comprises a transparent object.
 9. The 3D sensing deviceaccording to claim 8, wherein a material of the transparent objectcomprises glass.
 10. A specular reflection object detection methodcomprising: commanding a flood light source to emit a flood light and astructured light source to emit a structured light in sequence;commanding an image sensor to sense a first reflective light formed byan object reflecting the flood light and a second reflective lightformed by the object reflecting the structured light in sequence insequence, so as to obtain a first image frame corresponding to the firstreflective light and a second image frame corresponding to the secondreflective light; combining the first image frame and the second imageframe into a determination frame; and determining that the object is aspecular reflection object in response to determining that thedetermination frame has at least two spots having gray levels satisfyinga predetermined condition.
 11. The specular reflection object detectionmethod according to claim 10, wherein combing the first image frame andthe second image frame into the determination frame comprises:calculating difference values between gray levels of pixels of the firstimage frame and gray levels of corresponding pixels of the second imageframe; and calculating absolute values of the difference values torespectively form gray levels of pixels of the determination frame. 12.The specular reflection object detection method according to claim 10,wherein the predetermined condition is that a gray level is greater thana first threshold value.
 13. The specular reflection object detectionmethod according to claim 12, wherein determining that the determinationframe has at least two spots having gray levels satisfying thepredetermined condition comprises: dividing the determination frame intoa plurality of regions; finding regions having an average of gray levelsless than a second threshold value; and determining that thedetermination frame has at least two spots having gray levels satisfyingthe predetermined condition in response to determining the number ofspots having gray levels greater than the first threshold value isgreater than or equal to a third threshold value.
 14. The specularreflection object detection method according to claim 13, wherein thethird threshold value is
 2. 15. The specular reflection object detectionmethod according to claim 13, wherein the regions are arranged in arectangular array.
 16. The specular reflection object detection methodaccording to claim 10, wherein the at least two spots are contributed bythe specular reflection object specularly reflecting the flood light andthe structured light.
 17. The specular reflection object detectionmethod according to claim 16, wherein the specular reflection objectcomprises a transparent object.
 18. The specular reflection objectdetection method according to claim 17, wherein a material of thetransparent object comprises glass.